Answer Extraction Towards Better Evaluations Of NLP Systems
نویسندگان
چکیده
We argue that reading comprehension tests are not particularly suited for the evaluation of NLP systems. Reading comprehension tests are specifically designed to evaluate human reading skills, and these require vast amounts of world knowledge and common-sense reasoning capabilities. Experience has shown that this kind of full-fledged question answering (QA) over texts from a wide range of domains is so difficult for machines as to be far beyond the present state of the art of NLP. To advance the field we propose a much more modest evaluation set-up, viz. Answer Extraction (AE) over texts from highly restricted domains. AE aims at retrieving those sentences from documents that contain the explicit answer to a user query. AE is less ambitious than full-fledged QA but has a number of important advantages over QA. It relies mainly on linguistic knowledge and needs only a very limited amount of world knowledge and few inference rules. However, it requires the solution of a number of key linguistic problems. This makes AE a suitable task to advance NLP techniques in a measurable way. Finally, there is a real demand for working AE systems in technical domains. We outline how evaluation procedures for AE systems over real world domains might look like and discuss their feasibility. Answer Extract ion Towards bet ter Evaluat ions of N L P Systems R o l f S c h w i t t e r and D i e g o M o l l ~ and R a c h e l F o u r n i e r and M i c h a e l H e s s D e p a r t m e n t of In fo rma t ion Technology C o m p u t a t i o n a l Linguist ics G r o u p Univers i ty of Zurich CH-8057 Zurich [schwitter, molla, fournier, hess] @ifi. unizh, ch
منابع مشابه
A New Method for Improving Computational Cost of Open Information Extraction Systems Using Log-Linear Model
Information extraction (IE) is a process of automatically providing a structured representation from an unstructured or semi-structured text. It is a long-standing challenge in natural language processing (NLP) which has been intensified by the increased volume of information and heterogeneity, and non-structured form of it. One of the core information extraction tasks is relation extraction wh...
متن کاملIdentifying Expressions of Opinion in Context
While traditional information extraction systems have been built to answer questions about facts, subjective information extraction systems will answer questions about feelings and opinions. A crucial step towards this goal is identifying the words and phrases that express opinions in text. Indeed, although much previous work has relied on the identification of opinion expressions for a variety...
متن کاملEmpirical Methods in Information Extraction
Most corpus-basedmethods in natural language processing (NLP)were developed toprovide an arbitrary text-understanding application with one or more general-purpose linguistic capabilities. This is evident from the articles in this issue of AI Magazine. Charniak and Ng/Zelle, for example, describe techniques for part-of-speech tagging, parsing, and word-sense disambiguation. These techniques were...
متن کاملLanguage Learning: Beyond Thunderdome
Remember: no matter where you go, there you are. The eight years from 1988 to 1996 saw the introduction and soon widespread prevalence of probabilistic gen-erative models in NLP. Probabilities were the answer to learning, robustness and disambiguation, and we were all Bayesians, if commonly in a fairly shallow way. The eight years from 1996 to 2004 saw the rise to preemi-nence of discriminative...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
دوره شماره
صفحات -
تاریخ انتشار 2000